Skip to content

Clarify BIDS dataset organization and distribution considerations #688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Clarify BIDS dataset organization and distribution considerations #688

wants to merge 8 commits into from

Conversation

@yarikoptic
Copy link
Contributor

@yarikoptic yarikoptic commented Aug 7, 2025 •
edited
Loading

  • Add note about BIDS Raw datasets being distributable without derivatives
  • Include dataset_description.json in directory structure examples
    to emphasize where we observe legit BIDS datasets
  • Explain disadvantages of nested dataset organization for distribution
  • Clarify that sourcedata can contain Raw, non-BIDS, or derivative datasets
  • Add requirement for BIDSVersion key to identify BIDS datasets in subdirectories
  • Adjust example of non-nested dataset organization in my_study folder

This PR is on top of the

by @kabilar

Potential TODOs but may be after in a separate PR:

  • Add "study" DatasetType (should be added in 1.11.0, or will it be @1.10.1 @effigies ?)

@yarikoptic yarikoptic mentioned this pull request Aug 7, 2025
Closed
kabilar
kabilar reviewed Aug 7, 2025
View reviewed changes
Copy link
Member

@kabilar kabilar left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @yarikoptic, thanks for putting these changes together as it clarifies several pieces for me.

cc @satra

kabilar
kabilar reviewed Aug 7, 2025
View reviewed changes
@kabilar
Copy link
Member

kabilar commented Aug 7, 2025 •
edited
Loading

The following examples show three ways to organize, relative to each other,
a raw BIDS dataset, a preprocessed derivative dataset, and an analysis that uses both as inputs.

We may also need to change the above statement to more accurately reflect the examples provided. Perhaps to the following:

The following examples show three ways to organize datasets:

  1. A BIDS Raw dataset with derivative preprocessed and analysis data.
  2. A BIDS Derivative dataset which uses both raw and preprocessed data as inputs.
  3. A BIDS dataset with raw and derivative data.

kabilar
kabilar reviewed Sep 8, 2025
View reviewed changes
kabilar and others added 3 commits October 2, 2025 13:24
@kabilar @effigies
Remove example of rawdata/ at the top level of a BIDS datasets
775dcb5
@yarikoptic @effigies
Clarify BIDS dataset organization and distribution considerations
a66ee29 
- Add note about BIDS Raw datasets being distributable without derivatives
- Include dataset_description.json in directory structure examples
  to emphasize where we observe legit BIDS datasets
- Explain disadvantages of nested dataset organization for distribution
- Clarify that sourcedata can contain Raw, non-BIDS, or derivative datasets
- Add requirement for BIDSVersion key to identify BIDS datasets in subdirectories
- Re-Include example of non-nested dataset organization in my_study folder
  (I based this change on top of the removal proposal in
  bids-standard#687)
@yarikoptic @kabilar @effigies
clarify about nesting
32e5edb 
Co-authored-by: Kabilar Gunalan <[email protected]>
@effigies effigies closed this Oct 2, 2025
@effigies effigies reopened this Oct 2, 2025
effigies
effigies reviewed Oct 2, 2025
View reviewed changes
effigies
effigies reviewed Oct 2, 2025
View reviewed changes
Copy link
Contributor

@effigies effigies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT?

julia-pfarr
julia-pfarr requested changes Oct 2, 2025
View reviewed changes
Copy link
Member

@julia-pfarr julia-pfarr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I don't want to initiate a new discussion (or this discussion again...) but I kind of don't understand why we need to serve all these different cases of where to store what?

Why can't we just say

  • root/sourcedata --> out of scanner data
    root/raw --> raw data in BIDS
    root/derivatives --> anything starting with preprocessing, in BIDS
  • If you have only one of those three, adding a second level folder is not necessary.

Can someone please explain to me why it is not possible to do this? I'm clearly missing something and I would like to understand...sorry!

@yarikoptic
Copy link
Contributor Author

yarikoptic commented Oct 2, 2025

@julia-pfarr a quick answer on "Why can't we just say" is that because it is not what the standard is permitting/expecting ATM.

This repository is for the website which just provides a little more of "expanded for a user re-digestion of the specification". If we start saying here something not allowed by the specification, we would do disservice IMHO.

Any decisions to change things up should be discussed against specification first.

@yarikoptic @effigies @julia-pfarr
Tune up sentences about nesting
1be101e 
TODO: add generic phrasing based on what @effigies suggested

Co-authored-by: Chris Markiewicz <[email protected]>
Co-authored-by: Julia-Katharina Pfarr <[email protected]>
yarikoptic
yarikoptic commented Oct 2, 2025
View reviewed changes
@julia-pfarr
Copy link
Member

julia-pfarr commented Oct 2, 2025

@julia-pfarr a quick answer on "Why can't we just say" is that because it is not what the standard is permitting/expecting ATM.

This repository is for the website which just provides a little more of "expanded for a user re-digestion of the specification". If we start saying here something not allowed by the specification, we would do disservice IMHO.

Any decisions to change things up should be discussed against specification first.

Ok, I see, that makes sense, thanks! So I understand from your answer that my suggestion is technically possible and not totally unreasonable, this is just not the right place for it.

@julia-pfarr julia-pfarr mentioned this pull request Oct 8, 2025
Open
kabilar
kabilar reviewed Oct 9, 2025
View reviewed changes
@kabilar
Copy link
Member

kabilar commented Oct 9, 2025

Hi @yarikoptic, the conversation above was resolved, but I don't think the following update to the text has been made.

Original:

The following examples show three ways to organize, relative to each other,
a raw BIDS dataset, a preprocessed derivative dataset, and an analysis that uses both as inputs.

Suggestion:

The following examples show three ways to organize datasets:
## 1. BIDS Raw with derivatives/
## 2. BIDS Derivative with sourcedata/raw
## 3. BIDS Study with sourcedata/raw and derivatives/

julia-pfarr
julia-pfarr reviewed Oct 15, 2025
View reviewed changes
@julia-pfarr
Copy link
Member

julia-pfarr commented Dec 1, 2025

@yarikoptic I fixed my suggestion, wdyt? Should I merge or do you want to change something?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@effigies effigies effigies left review comments

@julia-pfarr julia-pfarr julia-pfarr approved these changes

+1 more reviewer

@kabilar kabilar kabilar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yarikoptic @kabilar @julia-pfarr @effigies